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[57] ABSTRACT 

A system for automatically detecting and recognizing the 
identity of a de format) le object such as a human face, within 
an arbitrary image scene. The system comprises an object 
detector implemented as a probabilistic DBNN, for deter- 
mining whether the object is within the arbitrary image 
scene and a feature localizer also implemented as a proba- 
bilistic DBNN, for determining the position of an identify- 
ing feature on the object such as the eyes. A feature extractor 
is coupled to the feature localizer and receives coordinates 
sent from the feature localizer which are indicative of the 
position of the identifying feature and also extracts from the 
coordinates information relating to other features of the 
object such as the eyebrows and nose, which are used to 
create a low resolution image of the object. A probabilistic 
DBNN based object recognizer for determining the identity 
of the object receives the low resolution image of the object 
inputted from the feature extractor to identify the object, 

13 Claims, 6 Drawing Sheets 
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NEURAL NETWORK FOR LOCATING AND 
RECOGNIZING A DEFORMABLE OBJECT 

FIELD OF INVENTION 

The present invention relates generally to machine vision 5 
and more particularly, to a system which implements 
decision-based neural networks that operate to locate and 
recognize deformable objects such as the human face. 

BACKGROUND OF THE INVENTION 10 

The task of detecting and recognizing a deformable 
pattern or object is an important machine learning and 
computer vision problem. The task involves finding and 
identifying a specific but locally deformable pattern in an is 
image, such as a human face. Machine learning and com- 
puter vision has many important commercial applications. 
Such applications include but are not limited to ATM, access 
control, surveillance, and video conferencing. Accordingly, 
machine learning and computer vision has attracted much 20 
attention in recent years. 

Face recognition systems used in person identification, 
typically employ a face detector which determines the 
location and extent of one or more human faces in a 
non-uniform arbitrary image scene. Such systems find this 25 
task difficult because human faces are naturally structured 
and made up of deformable components such as the cheeks, 
the mouth, the forehead, etc. In any case, once the face has 
been located, the system then compares the face to other 
faces stored in a database in order to identify the person. 30 

For systems used in many visual monitoring and surveil- 
lance applications, it is important that the system be capable 
of determining the position of the human eyes from an image 
or an image sequence containing a human face. Once the 
position of the eyes is determined, all of other important 35 
facial features, such as the positions of the nose and the 
mouth, can be determined. This information can then be 
used for a variety of tasks, such as to recognize a face from 
a given face database 

40 

The key issue and difficulty in face detection is to account 
for the wide range of allowable facial pattern variations in a 
given image scene. In the past, there have been three main 
approaches for dealing with these pattern variations, 
namely: (1) the use of correlation templates, (2) spatial 45 
image invariants, and (3) view-based eigen-spaces, etc. 

Correlation templates compute a similarity measurement 
between a fixed target pattern and the candidate image 
location. If the output exceeds a certain threshold, then a 
match is confirmed, i.e., a face detected. There are some face 50 
detection systems that use a bank of several correlation 
templates to detect major facial subfeatures in an image 
scene. However, the performance of such systems is limited 
in that the class of all potential face patterns is too varied to 
be modeled by a simple bank of correlation templates. ss 

Spacial image-invariance schemes assume that some 
common and unique spatial image relationships exist in all 
face patterns. Such a set of image invariants must be 
checked for positive occurrences of these invariants at all 
image locations. One particular image-invariance scheme go 
for example, is based on the local ordinal structure of 
brightness distribution between different parts of a human 
face. 

Avery closely related approach to correlation templates is 
that of view-based eigen-spaces. This approach assumes that 65 
the set of all possible face patterns occupies a small and 
easily parameterizable sub-space in the original high dimen- 
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sional input image vector space. Typically, the approach 
approximates the subspace of face patterns using data clus- 
ters and their principal components from one or more 
example sets of face images. An image pattern is classified 
as "a face" if its distance to the clusters is below a certain 
threshold, according to an appropriate distance metric. 
However, this approach has only been demonstrated on face 
images in substantially uniform backgrounds. 

There are algorithms and techniques which presently exist 
for eye localization are generally based on Hough transform, 
geometry and symmetry check, and deformable models. 
Most of these algorithms and techniques are generally 
inadequate against shape changes, and are time consuming. 
Furthermore, none of these existing methods can locate eyes 
when the eyes are closed. 

Neural network models have been found to be very 
amenable to face recognition systems. As is well known in 
the art, a neural network is generally an implementation of 
an algorithm which enables a computer to be adaptive by 
learning directly from inputted data which is used to "train" 
the computer to perform a certain task. This enables such a 
computer to process data that only somewhat resembles the 
training data. Moreover, such computers are also capable of 
processing incomplete or imperfect data or providing a 
measure of fault tolerance. Additionally, such computers can 
recognize complex interactions among the input variable of 
a system. Since neural networks are parallel, a large network 
can achieve real-time speeds making their application more 
practical in many areas. 

A neural network is generally comprised of intercon- 
nected computational elements or units which operate in 
parallel and are arranged in patterns which broadly mimic 
biological neural networks. Each connection between com- 
putational elements is associated with a modifiable weight. 
In operation, a computational element converts a pattern of 
incoming signals into a single outgoing signal that it sends 
to other connected computational elements. It does this by 
multiplying each incoming signal by the weight on the 
connection and adds together all the weighted inputs to get 
a quantity called the total input. Then, the computational 
element uses an input-output function that transforms the 
total input into an outgoing signal. In order for the neural 
network to perform a specific task, the computational ele- 
ments must be connected to each other in certain network 
arrangement, and the weights must be set appropriately. The 
connections determine how the computational elements will 
influence each other and the weights determine the strength 
of this influence. 

It is, therefore, an object of the present invention to 
provide a decision-based neural network and system for 
implementing the network that locates and recognizes 
deformable objects with specific applications directed at 
detecting human faces and locating eyes in the faces. 

SUMMARY OF THE INVENTION 

A system for automatically detecting and recognizing the 
identity of a deformable object such as a human face, within 
an arbitrary image scene. The system comprises an object 
detector implemented as a probabilistic DBNN, for deter- 
mining whether the object is within the arbitrary image 
scene and a feature localizer also implemented as a proba- 
bilistic DBNN, for determining the position of an identify- 
ing feature on the object such as the eyes. A feature extractor 
is coupled to the feature localizer and receives coordinates 
sent from the feature localizer which are indicative of the 
position of the identifying feature and also extracts from the 
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coordinates information relating to other features of the 
object such as the eyebrows and nose, which are used to 
create a low resolution image of the object. A probabilistic 
DBNN based object recognizer for determining the identity 
of the object receives the low resolution image of the object 
inputted from the feature extractor to identify the object. 

Also provided in the present invention is a method for 
automatically detecting and recognizing the identity of a 
deformable object within an arbitrary image scene. In the 
method, the image scene is preprocessed into subimages. 
Each of the subimages are compared with an object detector 
database which stores different versions of the object in 
order to determine whether any of the subimages is the 
object. The coordinates of an identifying feature on the 
object are then located by comparing the coordinates with a 
feature localizer database which stores coordinates indica- 
tive of different versions of the identifying feature. Infor- 
mation relating to other features of the object are extracted 
from the coordinates to create a low resolution image of the 
object. Next, the low resolution image of the object image is 
inputted into an object recognizer and the identity of the 
object is made. 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a detailed understanding of the present invention, 
reference should be made to the following detailed descrip- 
tion taken in conjunction with the accompanying drawings 
wherein: 

FIG. 1 is a diagrammatic view of an exemplary embodi- 
ment of the face locating and recognition system of the 
present invention; 

FIG. 2A is a schematic diagram of a DBNN; 

FIG. 2B is a structural depiction of a probabilistic DBNN 
according to the present invention; 

FIG. 3 is a schematic diagram of a probabilistic DBNN 
according to the present invention; 

FIG. 4 is a diagrammatic view of second exemplary 
embodiment of the face locating and recognition system of 
the present invention which includes a face verifier; and 

FIG. 5 is schematic diagram of a multi-channel DBNN 
according to the present invention. 

DETAILED DESCRIPTION OF TOE 
INVENTION 

Although the present invention can be used to locate most 
any deformable pattern or object, the present invention is 
especially suited for use in face detection, eye localization 
and person identification. Accordingly the present invention 
will be described in this context. 

Face detection, eye localization, and face recognition are 
essentially pattern classification problems. For example, in 
face detection a given pattern is classified into two classes, 
face or non-face. In the present invention, a probabilistic 
variant of a decision-based neural network (DBNN) is 
provided to perform this classification task. More 
specifically, both face detection and eye localization are 
implemented by a probabilistic DBNN which will be 
described later on in greater detail. For these applications, 
and more generally for any deformable pattern detection, 
there is only one subnet required in the DBNN. In the 
exemplary embodiment of the present invention, the subnet 
represents the face/eye class. 'Rms, for an input pattern x, if 
the discriminant function value of the subnet is larger than 
the threshold, then x is recognized as a face/eye. Otherwise, 
x is considered as a non-face/eye. 
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Referring now to FIG. 1, an exemplary embodiment of a 
fully automatic face recognition system of the present inven- 
tion is shown and designated by numeral 10. The system 10 
comprises a video camera 12 for inputting an arbitrary 

5 image scene 11 with 320 by 240 pixels. A DBNN-based face 
detector 14 is coupled to the video camera 12 and includes 
a memory 16 which operates as a database for storing 
images of different human faces. The face detector 14 
determines whether a face is within the arbitrary image 

10 scene 11. The data stored in the face database 16 is used to 
train the face detector 14. During training, updated network 
weighting parameters and thresholds are stored in the face 
database 16. 

The input images from the video camera 12 are first 
15 preprocessed before inputting to the DBNN-based face 
detector 14. The inputs to the DBNN-based face detector 14 
are a set of images with predefined coordinates. To detect a 
face in an input image, each of the possible subimages is 
processed to see if it represents a face. A confidence score is 
20 produced, indicating the system's confidence on this detec- 
tion result. If the score is below some threshold, then no face 
is detected. 

If positive identification of a face is made by the face 
detector 14, a DBNN-based eye localizer 18 which is 

25 coupled to the face detector 14, is activated to locate both 
eyes in the face image. Knowing the exact position of the 
eyes provides a very effective means for normalizing the 
face size and reorienting the face image. The pattern reso- 
lution used for the eyes is substantially higher than that used 

30 for the faces. Both the face detector 14 and the eye localizer 
18 are insensitive to small changes in the head size, the face 
orientation (up to approximately 30%), and the presence of 
eye glasses. 

35 The eye localizer 18 also includes a memory 20 which 
operates as a database for storing information pertaining to 
the coordinates of various different eyes. The eye localizer 
18 determines the coordinates of each eye and then sends 
these coordinates to a facial feature extractor 22 as will be 

40 explained later below. The data stored in the eye database 20 
is used to train the eye localizer 18. During training, updated 
network weighting parameters and thresholds are stored in 
the eye database 20. 
The facial feature extractor 22 is coupled to the eye 

45 localizer 18 and uses the eye coordinates sent from the eye 
localizer 18 to extract a low resolution subimage which is 
approximately 140 by 100 pixels and corresponds to the face 
region. The facial region contains the eyebrows, the eyes, 
and the nose (excluding the mouth). Such a facial region 

50 yields a very high degree of confidence in that it offers 
stability against different facial expressions, hair styles, and 
mouth movement. Improved classification can also be 
gained from secondary facial features such as the hairline 
and the mouth. 

55 The facial feature extractor 22 normalizes the intensities 
and the edges in the facial region (to a range between 0 and 
1) to compensate for changing illumination. Edge filtering 
and histogram modification techniques can be applied to 
recondition the facial images. The normalized and recondi- 

60 tioned images of 140 by 100 pixels are then reduced to 
coarser feature vectors of approximately 13 by 9 pixels. 
Adopting lower resolution facial features provides substan- 
tial reductions in computational cost and storage space and 
increases the tolerance on face/eye location errors. 

65 In order to assure sufficient diversity of real facial images 
in the training set, the algorithm takes the acquired sensor 
image and transforms it to create additional training exem- 
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plars otherwise known as virtual training patterns. Two 

kinds of training patterns are used. The first training pattern . 

consists of positive patterns (face/eye patterns) which are ' w t ' ^ ^ 

used for reinforced learning. The second training pattern Accordingly, the winning class for the pattern is the jth 

consists of negative patterns (non-face/eye patterns) which 5 class ( subnet ). when and only when j*i , (i.e., when x (m) is 

are used for antireinforced learning. The network weighting misclassified), the following update will be performed: 
parameters and the thresholds are updated by this reinforced/ 

antireinforced learning. A more detailed discussion of virtual Reinforced Learning: w/-* 1 W,«">+n w t ) 

pattern generation will be explained later below. Antireinforced Learning: v^W/">-t,V<K*, m>,) (2) 

The feature vector generated by the facial feature extrac- 10 

tor is then fed into a DBNN-based face recognizer 24 for Typically, one output node is designated to represent one 

recognition. The face recognizer 24 includes a memory 26 class. The AU-Class-in-One-Network (ACON) structure is 

which operates as a database for storing person identification adopted by in a conventional MLP, where all the classes are 

information lumped into one super-network. The supernet has the burden 

Hie trained system can be easily adapted to a face 15 of hfving simultaneously satisfy all the teachers so the 

verification application. Due to the distributed structure of a number °f iinite K tends to be large Empu^cal results 

DBNN, any individual person's database may be individu- have confirmed that the convergence rate of ACON degrades 

ally retrieved for verification of his or her identity as drastically with respect to the network size because the 

proclaimed training of hidden units is influenced by potentially conflict - 

T^xr^T . . . , . j • ,i c j < . 20 ing signals from different teachers. 

The DBNN techniques implemented in the face detector n e • ♦ r T r >*a u a- ~ ~ „u™,v,, T tu„ 

iL , i- -go j *u r • Re ferrmg to FIG. 2 A, a schematic diagram showing the 

14, the eye localizer 18 and the face recognizer 24 as qbnn used in the present invention designated by the 

desenbed .n the system 10 above can be applied in other 3Q ^ shoy £ ^ DBNN 3Q a 0ne . class . 

sumlar systems for detecting virtually any type of deform- in . 0ne . Network (0C0N) structure> where one subnet ^ 

able object. The discussion which follows immediately . . , , , , i c u „i™t 11 -xa ™a it 

, J , T^r.xTxi • 1 . j ■ *u 25 designated to one class only. Each subnet 32, 34 and 30 

below details the probabilistic DBNN implemented in the ? . . . , . f , f mt l antUorv M 

. r r specializes in distinguishing its own class from the others, so 

above-described system. the number of hidden units is sma n. Experimental 

The DBNN used in the present invention uses a distnb- results based Qn a broad range of ap pii ca tions (OCR, speech, 

uted and localized updating rule based on reinforced and and face recognition ) suggest that 3-5 hidden units per 

anti-reinforced learning strategy. The gradient of the dis- 3Q subnet afe preferred 7^ 0 CON structure of a DBNN 

enminant function with respect to the weight parameters is makeg {{ mQS{ suitable for incremental training, i.e., network 

used as an updating direction. The mam merit of this is that upgrading up0D adding/removing memberships, 

it enables the border between any two classes to be settled ^ trainirig scheme of the DBNN 30 is based on Locally 

mutually, with minimum side-effects on other borders. In the Unsupervised Globally Supervised (LUGS) learning. There 

DBNN, the teacher only tells the correctness of the classi- 35 are lwo phases in this scheme : during the locally- 

fication for each training pattern. The teacher is a set of unsuper vised (LU) phase, each subnet is trained 

symbols, t={i,}, which label the correct class for each input individually, and no mutual information across the classes 

pattern. Unlike an approximation formulation, exact values may be utilized> MiGT the LU phase is completed, the 

of the teachers are not required. Accordingly, the objective tra i ning enters tne Globally-Supervised (GS) phase. In GS 

of the training is to find a set of weights which yields a 4Q phase teacher i n f ormation is introduced to reinforce or 

correct classification. anti- reinforce the decision boundaries obtained during LU 

For complex pattern distribution, the discriminant func- phase. The discriminant functions in all clusters will be 

tion is usually a priori unknown. This leads to a credit trained in a two-phase learning technique comprising a 

assignment rule on when, what, and how to perform network global level and a local level. In the global level phase of 

updating. Its main purpose is to alleviate the problem of 45 learning, a supervised mutual (decision-based) learning rule 

overtraining the networks. There are three main aspects of ^ adopted. In the local level phase of learning, initialization 

the training rule: when to update; what to update; and how ^ al wa y S by an unsupervised clustering method, such as a 

to update the weights. k-mean. If too many clusters are adopted, overfitting can 

Under the training rule, knowing when to update is result, which in turn will hamper the generalization capa- 

determined by a selective training scheme for example, 50 bility. A proper number of clusters can be determined by an 

which updates the weight only when there is misclassifica- unsupervised clustering technique. 

tion. Since the rule is distributive and localized, knowing The learning rule of the DBNN 30 is very much decision- 

what to update is accomplished by applying reinforced boundary driven. When pattern classes are clearly separated, 

learning to the subnet corresponding to the correct class and such learning usually provides very fast and yet satisfactory 

anti-reinforced learning to the subnet corresponding to non- 55 learning performance. Application examples including OCR 

correct class. Updating under the rule is accomplished by and (finite) face/object recognition. Different tactics are 

adjusting the boundary by updating the weight vector w needed when dealing with overlapping distribution and/or 

either in the direction of the gradient of the discriminant issues on false acceptance/rejection, which arises in appli- 

function (i.e., reinforced learning) or opposite to that direc- cations such as face recognition and verification. In such 

tion (i.e., antireinforced learning). 60 applications, the present invention provides a probabilistic 

The following describes the Decision-Based Learning variant of the DBNN as described earlier in connection with 

Rule discussed immediately above: Suppose that S={x (1 \ . the face detector 14, eye localizer 18 and face recognizer 24 

. . , x (A °} is a set of given training patterns, each correspond- of the automatic face recognition system 10 of FIG. 1. 

ing to one of the M classes {to^-l, . . . , M}. Each class is Referring to FIG. 2B, an exemplary embodiment of a 

modeled by a subnet with discriminant functions, for 65 probabilistic DBNN denoted by the numeral 40 is schemati- 

example, (^(XjW,) i»l, ... ,M. Suppose that the m-th training cally shown. The subnets 42 and 44 of the probabilistic 

pattern x (m) is known to belong to class co, and; DBNN 40 are designed to model log-likelihood functions. In 



07/21/2004, EAST Version: 1.4.1 



5,850,470 

7 8 

the probabilistic DBNN 40, reinforced/antireinforced learn- bution with full-rank covariance matrix. A hyper-basis func- 

ing is applied to all the clusters of the global winner and the tion (HyperBF) is meant for this. However, for those appli- 

supposed (i.e. the correct) winner, with a weighting distri- cations which deal with high dimensional data but finite 

bution proportional to the degree of possible involvement number of training patterns, the training performance and 

(measured by the likelihood) by each cluster. 5 storage space discourage such matrix modelling. A natural 

The probabilistic DBNN 40 is designed to approximate simplifying assumption is to assume uncorrected features of 

the Bayesian posterior probabilities and likelihood func- unequal importance. That is, suppose that p(I|co ( , GJis a 

tions. It is well town that the optimal data classifier is the D . d i mea sional Gaussian distribution with uncorrected 

Bayes classifier. Suppose there are M classes {oa,, . . . , 03 M } features that is 

in the feature space, the Bayes decision network classifies JQ 

input patterns based on their posterior probabilities: Input x / \ (4) 

is classified to class co. if P(o) J x)>P(coJx), for all j^i. It can Q . 1 / 1 ? fa-**) 2 \ 

be shown that Bayes classifier has minimum error rate. D \ dml tfd ] 

Suppose the likelihood density of input x given class co,- (W 2 n \ rA 1 

is a D -dimensional Gaussian distribution, then posterior d 

probability P(cojx) can be obtained by Bayes rule: " u^, . . . , mjf is the mean vector, and 

*«**N diagonal malrk 

where P(co ( .) is the prior probability of class co, (by definition 20 is the covariance matrix. 

To approximate the density function in Eq. 4, we apply the 
M M elliptic basis functions (EBF) to serve as the basis function 

2 i>(o>;) = i), and 2 P(<a k )p{x\<a k ), for each cluster: 

25 i D i (5) 

The class likelihood function p(xK) can be extended to e ') = " — ^ ~T" ~ Wr <*¥ + Qr 

mixture of Gaussian distributions. Define p(x|co t -, Q r ) to be = rd 

one of the Gaussian distributions which comprise p(x|oo ( ), whcre 

R ™ D D 

P(jeK)= 2 Pie^pMui, G r ) 30 6 f o--^- ln2»- 2 lna^. 

r=l 2 d-1 



where Q^ifa, Xj denotes the parameter set for the cluster After an exponential transform, exp {co(x, co,-, B x )} can be 

x, P(© T |to J ) denotes the prior probability of cluster x when viewed the same Gaussian distribution as described in Eq. 4, 

input patterns are from class ai t -, and p(x|o),-, 0 x )»NtX, 35 except a minor notational change: In other 

By definition words, 

R 1 D (6) 

2 P{BM - 1. to;, e r ) - - 4 2 CO* - "rd) 2 + 

40 

. . . . , . , n/ , The learning rules for the probabilistic DBNN 40 is as 

In many applications it is > 'Prolate "> assume that f()llows , , he ^ Kheme for tfae mm tne LUGS 

=P(a),). Therefore the likelihood probability pix coj can . . , . c „ , T „ TT . j /TTA . 

replace posterior probability P(a,,.|x) to serve as discriminant P n ° c ' P ^°»°7 d n 1 ^Lo caU y} ,n f u P ervi f cl ( LU ) P has * 

function for the probabilistic DBNN can adopt several unsupervised 

"discriminant function of each subnet in the probabi- « learning schemes such as, LVQ, k-mean, EM as so forth. As 

listic DBNN models the log-likelihood function: for the Globally Supervised (GS) learning, the decision- 
based learning rule is adopted. Suppose that the m-th train- 

(3) m S pattern x< m) is known to belong to o) f ; and 



<«*, Wj) = logp^d)/) - log 



2 PtefofotriPr, (Oj) 



50 



tf/ m \ w^^ m \ wf m \ Vl*j (7) 



TTie overall diagram of such a discriminant function is ^ { ^ winnin class for tfae tem is lhe - dass 

depicted in FIG^2B which shows the structure of the } ^ and Qn when ( . when ^ fa 

probabilistic DBNN. The function node f( ) is a nonlineanty v . . 7 .,, JX . c „ / , / , _r ^ 

operator. If we make the assumption of PK>P(a>,), f( ) is niisdassificd), the foUowing update will be performed: 

a log operator (likelihood type). If P(co I .)^, f( ) is a 55 Rcinforccd (^^c-^v^ w j 
normalization operator. Its purpose is to make the discrimi- 
nant function approximate class posterior probability Antireinforced Learning: w/^W/^-tiV^*, wj) (8) 
(posterior type). The DBNN shown in the exemplary 

embodiment is of the likelihood type. Note that an explicit If the training pattern belongs to the so-called negative 

teacher value would not be required, although it is a super- 60 training (i.e. "unknown") set, then only the anti-reinforced 

vised training because the teacher's knowledge on correct learning rule will be executed — since there is no "correct" 

classification is crucial in the training. In FIG. 3, a schematic class to be reinforced. 

diagram of the probabilistic DBNN 40 for deformable object The gradient vectors in Eq. 8 are computed as follows: 
detection is shown. The action described in the parentheses 

is for the learning phase only. 65 afo h>,-) (9) 

In the most general formulation, the basis function of a a Wnl " ***** ~ 
cluster should be able to approximate the Gaussian distri- 
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-continued tern be included in the negative training set. When 

training the probabilistic DBNN, positive patterns are used 
*K*> *i) / l v \ for reinforced learning, and negative patterns are used for 

2 \ cw "^"^ j anti-reinforced learning. 

5 The third aspect is the generation of run-time negative 
where pattern. During the training phase, the probabilistic DBNN, 

P(e r )a))p(jr|ci)i ©,) while still under training, can be used to examine the whole 

M*) ° ^ „ /Q , ' \ /i ' J I x image database every k epochs. If the network falsely 

detects a face (eye) somewhere in an image, then that 
particular subimage will be included into the negative train- 
P^Ja^ (and P^, if P(co>P((o / )) can be learned by the 10 m g se t. 
EM algorithm: At epoch j, Since the probabilistic DBNN is amenable to multi sensor 

classification, additional sensor information can be easily 
w PCQ^Q^/K^K e f ) (io) incorporated to improve the recognition performance. Two 

r = 2*P(Oj fc |a>i)0VCi<' B )|6> 1 , e*) approaches to multi-sensor classification are possible in the 

15 present invention. The first approach involves a hierarchical 
Pf0j(o)O + o = n/An s h®( m ) classification, where sensor informations are cascaded in 

^ m= l T sequential processing stages. Possible candidates for hier- 

archical sensor are the hairline and the mouth. The second 
p(o)-)(M> = (i/jv) 2 p(a>-[x< ffl ))0) approach involves a multi-sensor fusion, where sensor infor- 

mal ' 20 mations are laterally combined to yield improved classifi- 

cation. 

o* *u u u'r *• T"\"DXTxr Afk * j ^u~u'u~*: n Referring to FIG. 4, a hierarchical information processing 

Since the probabilistic DBNN 40 provides probabihshc ^ on DBNN fc shown d P enoted b * 

outputs a similar procedure to the Neyman-Pearson hypoth- 5Q ^ £ 5Q ^ similaf tQ ^ m of nG 

esis is followed for threshold updating. Accordingly, testing x and toher a preprocessing module 52 wh ich 

is performed by setting a threshold to the network outputs provides hairline or mouth features. In FIG. 4, hairline 

and computing the probability of false acceptance and false features are provided by the preprocessing module 52. The 

rejection. In order to find out the most possible regions for hairline images are inputted to face verifier 54 along with 

patterns from class w,-, it is preferred to choose a threshold decisional information provided by the face recognizer 24. 

T f - so that an input x is classified to class co, if log p(x|o)>T ( -. Generally, this system operates by cascading two processing 

For an input x, if xGco, but log p(x|co / )<T I , then T, should 30 stages. More specifically, a face verifier stage is cascaded 

reduce its value. On the other hand, if x^cd,- but log after the (original) face recognizer stage. The face verifier 52 

p(x|a) ( )>T„ then T t should increase. An adaptive learning is itself is another DBNN classifier. Its function is to 

rule to train the threshold. T, is proposed in the present verify/reject the decision of the primary recognizer. The 

invention as follows: Define dsT-log p(x|a)). Also define a verifier can be trained by the decision-based learning rule. 

cost function 1(d). 1(d) can be either a step function, a linear 35 The input vector is a 12x8 feature vector obtained by 

function, or a fuzzy-decision sigmoidal function. Once the down-sampling the forehead/hairline region of the face 

network finishes the training, the threshold values can be image. Three verification schemes are possible in this sys- 

trained as follows: Given positive learning parameter t|, at tem * t „ t „ , , r . 

t . In the first scheme, recall that each subnet of the primary 

p J ' 40 DBNN recognizer generates a confidence score for an input 

T^~T^-r\l\d) if *eco, (reinforced learning) pattern. Suppose that the highest scorer is the i-th subnet. If 

the confidence score of subnet i is below the threshold, the 

Tp+^TP+^lXd) if x*mt (antireinforced learning) (11) tQp choice of me face ver ifi er checked. If the best match 

The following discussion details the technique used for of the forehead/hairline region is also class i then class i is 

generating training patterns for the probabilistic DBNN. 45 recognized and verified. Otherwise the test .pattern is deemed 

% 11 *u * * * «™ as an intruder s. If the confidence score of subnet l is above 

Generally, there are three mam aspects to the training pattern ^ and . f ^ . fe a ^ k ( k=6) 

generation scheme used in the present invention matches Qf ^ forehead/hairline re ^ 0D) the recogni tion is 

The first aspect is the generation of virtual training confirmed otherwise the person is rejected, 

patterns. In the beginning of the training phase, a certain ]n tfae secQnd similarity lists are introduced, 

number of facial images are selected to produce exemplar 50 £very dass faas [{s 0WQ similar i ty Ust Tbe list's lengths also 

face/eye patterns for training the earlier described DBNN- vary from person to person. Initially, the similarity list of 

based face detector and eye localizer. Typically, these exem- class j contains only class j itself. Assuming that the DBNN 

plar face/eye patterns can be extracted manually from those f ace verifier has now completed the training process by the 

images. For each exemplar pattern, virtual patterns can be decision-based learning rule. If a training pattern (originally 

generated by applying various afEne transformations such 55 from class j) is classified into another class, say, k, then the 

as, rotation, scaling, shifting and mirroring process to the class k will be added to the similarity list of class j. This 

original pattern. By this method, each of the exemplar process will be repeated until all the training patterns of the 

patterns is used to regenerate a number of up to 200 virtual known persons are tested. 

training patterns. In regard to the verification rule, if the confidence score 

The second aspect is the generation of positive/negative 60 of subnet i is below the threshold, the top choice of face 

training patterns. Not all virtual training patterns are con- verifier is checked. If the best match of the forehead/hairline 

sidered good face or eye patterns. If a virtual pattern is region is also class i, then class i is recognized and verified, 

slightly perturbed from the original exemplar pattern, it will Otherwise the test pattern is deemed as an intruder's. If the 

be included in the positive training set. This generally confidence score of subnet i is above the threshold, and if the 

enhances the robustness of the neural network. On the other 65 top one class of the face verifier is on the similarity list of 

hand, if the perturbation exceeds a certain threshold class i, the recognition is confirmed. Otherwise the person is 

(empirically established by trial-and-error), the virtual pat- rejected. 
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The third scheme is the same as the second scheme, At step j, 
except the confirmation is now made more strict. More 

specifically, if the confidence score of subnet i is below the ^ (ni) oftfr^KQ) ^ _ * ^ (m) ( 12 > 

threshold, its recognition may still be confirmed if, the top i, a^Vt^K Q) m=1 

choice iof the face ^verifier is class i and its confidence score 5 ^ ^ ^ fe { ^ ^ ^ fc ^ 

exceeds the threshold of the verifier. Otherwise the person femain during ^ retrieviQg pfaase 

will be rejected. If the confidence score of subnet i is above A more general version of mu i t i_channel fusion, referred 

the threshold, the recognition can be confirmed, only if the t0 ^ data-dependent channel fusion is also presented. 

top choice of the face verifier is on the similarity list of class 10 Instead of using the likelihood of observing x given a class 

i and its confidence score exceeds the threshold of the (p(x|o) 7 , Q)) to model the discriminant function of each 

verifier. Otherwise the person will be rejected. cluster, we shall use the posterior probabilities of electing a 

All three of these schemes substantially improve the ^^"^^"^^cStolSx^^^* 

performance of face recognizer. Experimental results have C , a . n ? e , nC . a ^L 000 eD< ? 1 1 u ' 

r t & , r , , , is which stands for the confidence on channel k when the input 

shown that the third scheme produces about the same (false m ^ x Accordmglv> the probab n it y m odel is also 

acceptance+false rejection) rate as the second scheme does. modified to become 
The difference between these two schemes is that the false 

rejection rate of the third scheme is higher while the false K (12A) 

acceptance rate is lower. 20 ^^V) - ^ P(c&)P(v> t \x, c k ) 

Referring to FIG. 5, an exemplary embodiment of a ^ ^ ^ caQ be obtained by p(a)Jx> C >I*a>JCJp(xK 

multi-channel DBNN for multi-sensor biometnc recognition c k )/v(x\C k ), and the confidence P(Cjx) can be obtained by 

is shown and denoted by the numeral 60. This approach the following equations: 

consists of several classifier channels CI, 1 -CI, 6 and C2,l- 25 

C2,6 each of which receives input vectors either from r(C^x) - p ^ p ^ c ^ ^ 12B - ) 

different sensors or from a portion of a higher dimensional 2iP(Ci)p(4ci) 

feature vector. Here, the channels C1,1-C1,6 and C2,l-C2,6 where p^c,,) can be computed straightforwardly by equa- 

are not differentiated into primary or secondary categories. t ^ on p^C^ 

Therefore, lateral fusion of information is more appropriate. 30 

The outputs of the channels are combined by some proper ^Pifa-^c^pip^a^ 

weightings W11-W26. The weighting factor is assigned , . , , 1 , \ 

• * tu. cj »u M u r and P(C>) can be learned by 12 (but replace p(xo).-, CJ with 

based on the confidence the corresponding channel has on its , ,-\ A *' , J , v . , . Li. 1 

1. c- i-m^xtxt . u u r *• p( x C*))- The term P(CJ can be interpreted as "the general 

recognition result. Since DBNN generates probabilistic ' C j „ . i_ 11 

& . . 1 . , ^ r confidence we have on channel k. 

outputs, it is natural to design the channel weightings to have 35 ^ ^ ^ ^ fa ^ OQ wei ^ u 

probability properties. C w is the output of the i-th subnet in Qeed tQ be computed for each testing pattern during the 

channel k, which is equal to p(x|a) C k ). Further note that the retrieving phase. 

confidence measure is W Jt/ »P(Cja) ( ) and that the combined 

output for subnet i is Oi, which is p(x|a) I ). TEST RESULTS 

40 

In class-dependent channel fusion, the weighting factors Experimental test results are briefly summarized below, 
correspond to the confidence P(Cjo) 7 ) for each channel. The probabilistic DBNN has consistently and reliably deter- 
Here P(Cjto / ) represents the indicator on the confidence on mined actual face positions, based on experiments per- 
channel k when the test pattern is originated from the w, formed on more than one thousand testing patterns. The 
class. (By definition, 45 probabilistic DBNN also yields very satisfactory eye local- 

ization performance. It is insensitive to small changes of the 
2 P(cAo>) 1 k eaQl si 26 * tne f ace orientation (up to approximately 30%), 

^ WO - and ^ p resence 0 f e y e gi asS es t The present invention is 

very robust against large variation of face features and eye 

50 shapes. The probabilistic DBNN takes only 200 ms on a 
so it has the property of a probability function.) Suppose that SUN Sparc2fJ workslation to find human faces in an image 

there are K channels in the subnet w„ and within each with 320x240 pixels. For a facial image with 320x240 

channel there are R clusters. The probability model of the pixels, the probabilistic DBNN takes 500 ms to locate two 

DBNN-based channel fusion network can be described as e yes on a SUN Sparc20 workstation. Furthermore, because 

follows: 55 of the inherent parallel and distributed processing nature of 

DBNN, the technique can be easily implemented via spe- 

pC*td - r PiQtopfa o ( " A) ciaUzed hardware for real lime Performance. 

The following is an example of the application 
performance, which was based on the experimental perfor- 
where p(x|co I , C*) is the discriminant function of subnet i in 6Q mances on public (FERE T) and in-house (SCR) databases, 
channel k, and p(x|a) i ) is the combined discriminant function Fil&u an experiment was conducted on 200-person (each 
for class <*>,. Note that x=[x, , . . . , x/] , and since p(xK, ^ib two front-views) of the ARPA/ARL FERET database. 
C*) is conditional on C*, only x k is involved in the above One image per person was used for training and the other for 
formula. After all the parameters within channels complete testing. A decision-boundary driven DBNN reached 100% in 
their training, channel confidence P(Cjco ( ) can be learned by 55 training accuracy and 96% in testing accuracy. An improved 
the following: Define a^P^a),). At beginning, assign probabilistic variant of the DBNN achieved 99% recognition 
ajt-l/K, Vk=l, . . . , IC rate. The SCR 80x20 database consists of 80 people of 
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different races, ages, and genders. The database contains 20 
images for each person. If a person wears glasses, 10 of the 
images are with glasses and 10 are without glasses. The 
training set comprised 4 images per person. The testing set 
included 16 images per person, 1280 images in total. For all 
of the images, the DBNN-based face detector always cor- 
rectly detected the center of the face thereby providing a 
100% success rate. Eye localization is a more difficult task 
than face detection, in particular when eye glasses are 
present. Among the 1280 images, the DBNN-based eye 
localizer mis-detected the eyes in 5 images by errors of more 
than 5 pixels. For the remaining 1275 images, the DBNN- 
based face recognizer achieved 100% recognition rate. An 
SCR-IM 40x150 database offered an opportunity to experi- 
ment with much larger orientation and other variations. The 
database contained 150 images for each of 40 persons. The 
images were acquired continuously while the person slowly 
moved and rotated his head. Head rotations were not only by 
very wide angle (up to 45 degrees) but also along various 
axes (i.e., left-right, up-down, and tilted rotations). The 
DBNN-based face detector and the DBNN-based eye local- 
izer worked correctly for 75% of the 6000 images in this 
database, which formed the so-called valid data set. A prior 
art face detector and eye localizer were trained only on 
frontal views. They could handle images up to only 30 
degree rotations. The DBNN-based face recognizer 
achieved a very high 98% recognition rate. 

The hierarchical DBNN-based face recognition system 
was tested with a 38-person face database. The hierarchical 
classification significantly reduces the false acceptance from 
9.35% to 0% and the false rejection from 7.29% to 2.25%, 
as compared to non-hierarchical face recognition. 

It should be understood that the embodiments described 
herein are merely exemplary and that a person skilled in the 
art may make many variations and modifications to these 
embodiments utilizing functionally equivalent elements to 
those described herein. Any and all such variations or 
modifications as well as others which may become apparent 
to those skilled in the art, are intended to be included within 
the scope of the invention as defined by the appended 
claims. 

We claim: 

1. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a decision -based neural network; 
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wherein said decision-based neural network comprises a 
one-class-in -one network structure having a plurality of 
subnets and a plurality of classes, wherein each one of 
said subnets is designated to one of said classes in order 

5 to distinguish it from said other classes; and 

wherein said decision-based neural network includes a 
training scheme having a first phase and a second 
phase, wherein said first phase includes individually 
training each of said subnets without mutually 

10 exchanging information between said classes and said 
second phase includes reinforcing learning and anti- 
reinforcing learning obtained during said first phase. 

2. The system according to claim 1, wherein said 
decision-based neural network comprises a probabilistic 

15 decision-based neural network, said reinforcing and anti- 
reinforcing learning being provided by a training pattern 
X c "° belonging to a class and: 

20 

wherein reinforced learning is performed according to: 
25 and antireinforced learning is performed according to: 

3. The system according to claim 1, wherein said 
decision-based neural network comprises a probabilistic 

30 decision-based neural network that includes a plurality of 
probabilistic outputs, each of which have a threshold value 
which is trained according to an adaptive learning rule: 

T^=TP~r\l'(d) if xeto,- (reinforced learning) 

35 

Tf^V-Tp+qlXd) ifxttoi (antireinforced learning). 

4. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 

4 0 scene, said system comprising: 

object detector means for determining whether said object 

is within said arbitrary image scene; 
feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
45 means being coupled to said object detector means; 
feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
50 information relating to other features of said object 
which is used to create a low resolution image of said 
object; 

object recognizer means for determining the identity of 
5S said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

60 wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
65 comprises a plurality of subnets, each of said subnets 
having a plurality of cluster basis functions which 
include cluster prior probabilities according to: 
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5. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 10 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 15 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 20 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision -based neural network 
comprises a plurality of subnets, each of said subnets 
includes a plurality of elliptic basis functions according 
to: 



including a discriminant function which comprises a 

nonlinearity operator. 
7. The system according to claim 6, wherein said dis- 
criminant function comprises a log operator which approxi- 
mates a log-likelihood function: 



25 



30 



35 
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6. A system for automatically detecting and recognizing 40 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- ^ 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 55 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said g 0 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 65 

wherein said probabilistic decision-based neural network 
comprises a plurality of subnets, each of said subnets 
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8. The system according to claim 6, wherein said dis- 
criminant function comprises a normalization operator 
which approximates a class posterior probability. 

9. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

further comprising object verifier means implemented as 
a probabilistic decision-based neural network for veri- 
fying the decision of said object recognizer means, said 
object verifier receiving additional information about 
said object which is cascaded in sequential processing 
stages in a hierarchical manner by said object verifier. 

10. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision -based neural 
network; and 
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wherein said probabilistic decision-based neural network 
comprises a plurality of classifier channels each having 
an output, said outputs being laterally fused by weight- 
ing said channels. 

11. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
comprises a plurality of classifier channels each having 
an output, said outputs being laterally fused by class- 
dependent channel fusion according to: 
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12. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature locator 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
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receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object; 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a probabilistic decision-based neural 
network; and 

wherein said probabilistic decision-based neural network 
comprises a plurality of classifier channels each having 
an output, said outputs being laterally fused by data- 
dependent channel fusion according to: 
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13. A system for automatically detecting and recognizing 
the identity of a deformable object within an arbitrary image 
scene, said system comprising: 

object detector means for determining whether said object 
is within said arbitrary image scene; 

feature localizer means for determining the position of an 
identifying feature on said object, said feature localizer 
means being coupled to said object detector means; 

feature extractor means coupled to said feature localizer, 
for receiving coordinates sent from said feature local- 
izer which are indicative of the position of said iden- 
tifying feature and for extracting from said coordinates 
information relating to other features of said object 
which is used to create a low resolution image of said 
object; and 

object recognizer means for determining the identity of 
said object, said object recognizer means being coupled 
to said feature extractor means and being operative to 
receive said low resolution image of said object input- 
ted from said feature extractor means to identify said 
object, 

wherein said object detector means, said feature localizer 
means, and said object recognizer means are each 
implemented in a decision-based one-class-in-one neu- 
ral network structure having a plurality of subnets and 
a plurality of classes, wherein each one of said subnets 
is designated to one of said classes in order to distin- 
guish it from said other classes, wherein said neural 
network includes a training scheme having a first phase 
and a second phase, wherein said first phase includes 
individually training each of said subnets without 
mutually exchanging information between said classes 
and said second phase includes reinforcing learning and 
anti-reinforcing learning obtained during said first 
phase. 
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